Temporal Action Segmentation with High-level Complex Activity Labels

The temporal action segmentation task segments videos temporally and predicts labels for all frames. Fully supervising such a model requires dense frame-wise annotations, which are expensive tedious to collect. This work is the first propose Constituent Action Discovery (CAD) framework that only video-wise high-level complex activity label as supervision segmentation. proposed approach automatically discovers constituent video actions using an classification task. Specifically, we define finite number of latent prototypes construct video-level dual representations with these learned collectively through training. setting endows our capability discover potentially shared across multiple activities. Due lack action-level supervision, adopt Hungarian matching algorithm relate ground truth semantic classes evaluation. We show can be extended from existing levels global level. global-level allows sharing activities, has never been considered in literature before. Extensive experiments demonstrate discovered help perform recognition tasks.